Skip to content

[Benchmark] Upgrade OmniDocBench to v1.5 - #1640

Open
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/omnidocbench-v1.5
Open

[Benchmark] Upgrade OmniDocBench to v1.5#1640
reacher-z wants to merge 1 commit into
open-compass:mainfrom
reacher-z:feat/omnidocbench-v1.5

Conversation

@reacher-z

Copy link
Copy Markdown

Summary

  • make the pinned 1,355-page OmniDocBench v1.5 TSV the default OmniDocBench dataset
  • retain the 981-page release as OmniDocBench_v1_0
  • parse v1.0 JSON and the published v1.5 Python-literal annotation format safely
  • pin both Hugging Face revisions and verify each download by MD5

Why

The v1.5 TSV already published in ouyanglinke/OmniDocBench_tsv cannot be evaluated by the current adapter: its answer column contains Python dictionary literals, so json.loads() rejects all 1,355 rows. The fallback here uses ast.literal_eval; it never executes annotation text.

Fixes #1456.

Validation

  • downloaded the pinned 1,506,235,920-byte v1.5 TSV; MD5: 995f1af5b4e24ad0a6417cbff708b3fc
  • parsed all 1,355 contiguous rows and checked page_info, layout_dets, and extra schemas
  • decoded and loaded rows 0, 677, and 1,354 (JPEG/JPEG/PNG)
  • exercised both end-to-end and table evaluator loading against a real v1.5 row: load_success=1, load_fail=0
  • perfect real-table smoke prediction: TEDS 1.0 and structure-only TEDS 1.0
  • 5 focused unit tests pass, including rejection of executable expressions
  • py_compile, git diff --check, and all configured pre-commit hooks pass

Implementation and validation were performed with OpenAI Codex assistance; I reviewed the resulting code, dataset provenance, hashes, and executed checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

【OmniDocBench】Recommend updating the benchmark version

1 participant